MY EXHIBITION: PERSONALISING UNENCUMBERED MULTIMEDIA CONTENT 
IN A MUSEUM ENVIRONMENT 



My Exhibition: Personalising Unencumbered Multimedia Content 
in a Museum Environment 

Prof. Chris Rust, (Sheffield Hallam University) 

Background 



' BARNES, C. and LILLFORD, 
S. R, 2007. Affective design 
decision making — issues and 
opportunities. CoDesign — 
Affective Communications 
in Design, Challenges for 
Researchers, 3(Supplement 1), 
pp 135-146. 

"DE PENNINGTON, N., 
et al. , 2007. Neuroiraaging of 
sensory and affective experience 
in the human brain. CoDesign 
— Affective Communications 
in Design, Challenges for 
Researchers, 3(Supplement 1), 
pp 45-55. 

^ HEALEY, R G. T. and LIGHT, 
A., 2007. Wlien scoring 
doesn't matter: the aesthetics 
of performance in arcade 
games. CoDesign- Affective 
Communications in Design, 
Challenges for Researchers, 
3(Supplement 1), pp. 91-96. 

^ Nanotechnology and 
the Running Shoe: Shock 
Revelation! , 2005. Documentary 
video. Directed by Richard 
JONES. Sheffield: University of 
Sheffield and Sheffield Hallam 
University Available at: <http:// 
www.nanoscience.dept.shef. 
ac.uk/images/gallery05/videol. 
php>. 



This project arose from an EPSRC research network, Affective 
Communication in Design, led by Prof. Tom Childs of University of 
Leeds, with co-investigators from Leeds, Sheffield, York and Sheffield 
Hallam Universities. In that two-year project we explored affect in 
exhibitions and products culminating with a conference that revealed a 
wide range of possibilities and positions. Some contributors focused on 
scientific approaches, for example affective engineering' and studies of the 
brain." Social research investigated affect as an aspect of performance and 
experience^ and practice-led research approaches explored affect as the 
outcome of creative practices in design and film-making.'' 

The My Exhibition research group includes specialists in engineering, 
human computer interaction, information studies, social anthropology, 
linguistics, design and media practice. It was a practice-led design project 
where practitioners from engineering, media arts and design collaborated 
to develop a practical vehicle to explore some implications of personalised 
ubiquitous computing. We also aimed to gain an understanding of how 
such an interdisciplinary research collaboration can work given the very 
diverse knowledge and expectations of the partners. 
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We set out to explore how people's experience of interactive media 
in public spaces might be 'personalised'. Behind this relatively simple 
statement lies a variety of assumptions, conditions and sub-texts. 

The location was a public museum, the Royal Armouries in Leeds. 
However we assumed that the work would be valid in anywhere that 
rich and complex material might be available to a variety of people with 
differing interests and inclinations. Our imagined audience would not 
be engaged in information-seeking, the 'goal-driven rational behaviour' 
assumed in many studies of interaction.' Instead we assumed that the 
audience might expect to be informed or entertained but have no explicit 
goals. Falk and Dirking' explain that some museum visitors are goal 
driven but others are not. We found that some of our visitors had expert 
interests in history or calligraphy, but others showed no evidence of any 
specific interest. 

During the project we adopted the idea of 'values' as a thinking tool 
althougli we did not engage with values as an issue in itself. Rather we 
used it as shorthand for ways of organising material and interests to reflect 
the different agendas that individuals and groups bring to a museum visit 
or other experience. 

The project has been concerned with the implications of future 
technologies. We were interested in working with large sets of images, 
sounds, speech and texts, and technologies now emerging will make it 
possible for such material to be presented in a variety of ways that are 
'unencumbered' by today's computing hardware. In theory at least, 
any surface might display dynamic images, sounds might be directed 
exclusively to any individual or group in a public space and the precise 
location, posture and identity of any individual within that space might 
be known at any moment. We replicated some of these functions, using 
existing techniques which provided usable, if limited, unencumbered 
interaction. 



Some research, for example Ehn and Kyng's low-fidelity mockups^ 
and 'Wizard of Oz' (WOz) methods where a human operator 'drives' 
the interface rather than a computer, has been successful without 
replicating future software and hardware. However, we were concerned 
with the visceral or affective qualities of interaction where timing and 
responsiveness are critical; a human operator cannot simulate the 
responsiveness of a computer system as seen in Dahlback et al's 
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discussion of WOz* and Dow et ij/'s' description of WOz in an interactive 
'visitor experience' application with location sensing and audio content. 
Thus it seemed important to create some interaction with real software as 
well as more detached scenarios. 

The research was driven also by an interest in personalisation or 
'mass customisation', important in manufacturing and in web design. 
For many years technology and industrial management has focused on 
individualised products or experiences,'"'" or designs that allow visitors 
to find personalised material on websites.'" While it is widely understood 
that consumers have diverse tastes and interests and marketeers have 
sophisticated ways to describe that,'^ we do not have equally sophisticated 
ways for consumers to shape experiences through direct action. 

One of our challenges was to provide tacit interaction but build on the 
experience of engineers designing systems to support explicit customer 
choices. Other research investigates how consumers might shape products 
for themselves," but we aimed to operate at a visceral, tacit level, allowing 
individuals to reveal interest without making explicit choices. This is 
only a partial aim; such systems can become oppressive and unethical, so 
participants must be aware of them and have the freedom to influence and 
subvert them, which implies some explicit choice and manipulation by 
users. Critics warn against oppressive use of 'spy' technologies'' but they 
might be enabling if people can control when and how they operate. 

We needed a design approach that used the different kinds of expertise 
and experience in the interdisciplinary group. Apart from creating 
a workshop environment to evolve our thinking and having people 
who were genuinely interested in each other's ideas, we exploited and 
developed scenario-based methods previously developed by members" 
using literary pastiches to introduce 'ready-made' characters and 
situations to design scenarios. We were also interested in the practical 
use of Web 2.0 techniques and this proved timely as some web-based 
tools for collaboration had reached a good level of development. In this 
inter-university project we used publicly available web resources to ensure 
sharing of planning, materials and knowledge. 



Finally we needed an arena which was provided in a timely and engaging 
way by the Royal Armouries Museum and their exhibition, with 
Prof. Peter Ainsworth of University of Sheffield, based on the mediaeval 
Chronicles of Froissart. Jean Froissart was a court poet and historian, 
connected with both sides of the 100 Years War, who gathered a rich 
and contemporaneous account of the history of the war and ordinary 
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people's experiences in it. His masterpiece of journalism became the 
text for illustrated manuscripts produced for rich clients by a Parisian 
entrepreneur, Pierre de Liffol. While Froissart's text was unvaried, each 
book was produced with an eye to the prejudices of the purchaser, 
influencing the visual narrative in the stylised illustrations. Like 
contemporary movies, the books were a product of a network of studios 
and master craftsmen, with similarly high budgets. 







Figure 1, Image from The 
Chronicles of Froissart, 
Bibhotheque Municipale 
Besanfon, ms 864, fol 150. 
Reproduced by permission of 
Bibhotheque Municipale Besant^on 



Peter Ainsworth and calligrapher/photographer, Colin Dunn, had 
photographed the chronicles in museums and libraries around the world. 
These, plus a Froissart manuscript loaned by Stoneyhurst College and the 
Royal Armouries' mediaeval arms and armour collection, provided the 
raw material for the exhibition, which included physical, graphical, 
audio-visual and interactive elements. Our project team created an 
introduction to the exhibition in a wide entrance corridor. 

Methods / Approach / Journey 



We had two main programmes of work; design and social research; but 
these overlapped and the design process became a more important aspect 
of the research than the study of visitors to the exhibition. 
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The core design group included systems engineers at Leeds University and 
an art/design/media group at Sheffield Hallam. Other project members 
were involved through a workshop series, running weekly during the most 
intensive design development. We started with a speculative, concept- 
building session, in a corridor that already contained 2D and 3D display 
materials to imagine how materials and techniques might be employed to 
create 'natural' interactions. For example we explored metaphors such as 
a torch to explore graphical material, triggering 'hot spots' to respond in 
various ways and we discussed/enacted how groups of individuals, each 
known to the system and with different agendas, might interact with each 
other to make sense of the interactions. 

A graphical storyboard then set out the main ideas, shared on the project 
wiki. We used characters from The Simpsons, easily recognised individuals 
with well-known agendas, as a tool for exploring design scenarios, in a 
tentative way in these storyboards which became our reference source of 
practical/technical design ideas. Later The Simpsons became full-blown 
members of the project team as described below. 



^^ Moore's law describes a 
long-term trend in the history 
of computing hardware, in 
which the number of transistors 
that can be placed on an 
integrated circuit has increased 
exponentially since 1962. Almost 
every measure of the capabilities 
of digital electronic devices is 
strongly linked to Moore's law, 
for example, processing speed 
and memory capacity'^ 

" WIKIPEDIA. 2009. Available 
at <http://en.wikipedia.org/wiki/ 
Moore%27s_law>. 



The design effort took on two themes: technology and content/interaction 
thinking. Technology assessment sought available technology for 
unencumbered interaction, embedding audio-visual materials 'naturally' 
in the environment controlled by personalised data. We were restricted 
by unexpected factors in the environment and technical limitations in the 
state of the art. On the other hand some rather vaguely-stated hopes for 
subtle interaction through posture and positioning became real elements. 
Moore's Law'' changed the project's technical plans several times with 
falling costs and improved performance in some technologies. 

Content and interaction development was also fluid and unpredictable, 
partly because it involved several disciplines. Tensions between the 
project's very ambitious, evolving aims and the priorities/limitations 
emerging in individual pieces of creative work were inevitable. We planned 
to exploit materials in the main exhibition, using ligliting, audio and other 
dynamic content to enhance the exhibits in personalisable ways. However 
the curators acquired other, explicit, interactive content which prevented 
us applying an interactive layer in the way we had expected. 



The alternative plan was to use a 2m wide entrance corridor for an exhibit 
that introduced key ideas about the exhibition. This gave more technical 
freedom to support the research, but it gave us the problem of creating 
new content from scratch. 
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An early stimulus for our personalisation strategy was recognising the 
distinct personalities and passions of people, historical and contemporary, 
who influenced the exhibition. The journalist's instincts of Jean Froissart, 
the glamour and pride of Sir John Chandos, a key figure in the war, the 
forensic insights of Karen Watts, Curator of Armour and the scholarly 
passion of Prof. Peter Ainsworth all illuminated the exhibition and 
seemed to represent different values that visitors might associate with. 
We identified a set of 7 working 'values', particular to this exhibition, to 
inform development of content and personalisation. 

These values did not define 7 separate types of visitor but might form 
the basis for a wide variety of profiles to help create individualised 
experiences. Cultural researchers, such as Hofstede and Bond" and Smith 
et aP° seek to identify relatively fixed frameworks of significant values 
that might predict the behaviour of people in communities. In contrast 
we used the term 'value' as a working concept, not related to any 
particular theory of values or culture, but simply shorthand for the 
particular focus or interest that an individual or group brought to the 
material in the exhibition. 
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Figure 2, Initial table of 
'values' developed in early 
review of content including 
an initial sketch of the value 
profile concept developed in 
the software 
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This led into development of content, initially through video production 
led by Jeff Baggott,( Sheffield Hallam University) with the Royal 
Armouries' actor/interpreters. As this material emerged with other 
members of the project group assisting or reviewing it, we realised that 
producing higli quality video was too time consuming to provide sufficient 
content for our needs with our resources. An audio content programme 
was therefore started, led by Mark Blythe, primarily a social researcher, 
trained and equipped by Jeff Baggott to create audio material. A variety 
of professional and amateur actors, acted scenes from the chronicles 
and experts provided commentaries that responded to our initial scheme 
of values. Subsequently this audio material formed the main interactive 
content, with the professional video illustrating what might be achieved 
in practice, allowing exploration of some technical possibilities. 



The material was coded, as ethnographic material might be coded using 
content analysis. Broad categorisations were agreed and an initial coding 
was then checked and modified with another coder (inter-coder agreement 
was high). This led to 7 new categories or 'values' (horror, comedy, 
passion, politics, daily life, personal experience and history). 
This review and restatement of our 'values' indicates a method that 
worked through iterations of a value scheme, refreshing it by examining 
the actual material as it emerged. This approach to what is essentially a 
'wicked problem' of evolving content, interaction scheme and technical 
underpinning in parallel has similarities to other current research'' 
where cycles of creative work and group reflection allow space for tacit 
interpretation by designers carrying out subsequent cycles of making 
(in this case the term 'designer' refers to all who make a creative 
contribution). This can be described"' using Polanyi's reasoning on tacit 
knowledge," with designers employing their tacit experience of the 
reflective discussion (Polanyi's 'proximal' element) in producing new 
explicit material (the 'distal' element). The approach also illustrates Rittel 
and Webber's'' process of resolving wicked problems: 



'an argumentative process in the course of ivhich an image of the problem 

and of the solution emerges gradually among the participants, 

as a product of incessant judgement, subjected to critical argument.' 
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Meanwhile the parallel exploration of technology had tested various 
techniques to capture position, identity and action of participants and 
to embed audiovisual material into the environment, simulating future 
technology Selective use of lighting was considered, using spotlights and 
backlighting to emphasise hotspots. 'Bare' liquid crystal display(LCD) 
screens were tested in configurations that might reveal visual content on 
demand but conceal the presence of the technology, similarly to theatrical 
ghost effects using differential lighting before and behind a gauze curtain. 

We also explored printing techniques and substrates to provide 'front' 
and 'rear' content on a printed display. Most techniques tested could be 
persuaded to work in isolation in perfect conditions, but not reliably 
enough to work in a semi-controlled environment. We also tested 
directional audio and found it very promising for targeting sound on 
hot spots in open spaces in the main halls of the museum, our confined 
corridor created too much reflection. 

Similar effects were found when attempting to locate badges that 
broadcast 'active radio-frequency identification (RFID)' signals. They 
were detected but in our confined space the intensity of the radio signals 
was not predictable enough for triangulation. RFID is widely used in 
interaction schemes but only where the user actively signals their presence, 
for example by 'swiping' a passive RFID card against a sensor. 

Eventually we compiled a technical plan that worked sufficiently for a 
public test of the personalisation scheme. 

In our entrance corridor, approximately 15m x 2m, we installed a false 
wall along one side. It appeared to be a conventional display of printed 
graphics showing images from the Froissart chronicles with some text 
about the exhibition. 

The wall concealed digital cameras, light boxes behind selective areas of 
images and text printed on the rear of translucent sections, loudspeakers 
and amplifiers and computers to control these devices and distribute 
audio content. The key to the interactions was a system of badges worn 
by visitors, each with a different colour to identify them. A computer 
analysed live camera images of the corridor to identify who was standing 
where in relation to the display which, had ten 500mm wide 'stations' 
along its length. 
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""^ An interactive programming 
environment for music, audio 
and media. 

"^ My Structured Query 
Language. A relational database 
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as a server providing multi-user 
access to a number of databases. 



This system was sensitive enough that a person could trigger the different 
stations to perform by swaying from side to side across the boundary 
between them, or could trigger the system to start or stop by turning 
towards or away from the wall, providing bodily interaction that miglit be 
become 'intuitive' in a short time. Initial evaluation with Royal Armouries 
staff indicated that this was a very engaging form of interaction and the 
exhibition curator was quick to become interested in its potential. 



Interaction needed a way of managing and personalising content, 
developed by the systems engineers. The concept of values led to the idea 
of value profiles, an array of linear scales indicating the visitor's interest 
in the different values we were using. At the start of the visit these could 
be derived from interviews or some explicit interactive method later 
modified by the system 'on the fly', as the visitor's actions revealed more 
or less interest. 

The software reflected our dual focus. The 'front end' used Max"VMSP 
(manage service provider) software to managed the sensors and 
audiovisual content, informing the MySQL" 'backroom' database system 
about the location and identity of visitors. This in turn instructed the 
front end what content to present while updating visitor profiles. The 
software mix reflected the interdisciplinary mix. 

Max/MSP is visual software widely used by artists to control hardware 
and content, used by an artist/programmer (James Brown). MySQL 
provides a more traditional database programming environment used 
by a software engineer (Pete Dawson), the two collaborating to build a 
complete system using MySQL to maintain a complex data structure and 
Max/MSP to control a diverse set of dynamic elements. 



Finally we needed to create the visitor's initial profile before they entered 
the exhibition, achieved by offering a way of moving forward and back 
through short animations with a physical dial to find a location in each 
animation that felt 'comfortable' to them, matching a point on one of the 
value scales. This was done in an explicit and humorous way, for example, 
an execution scene which became progressively more gory, introducing the 
visitor to the principles at work. However, having developed this tool, 
we will have the opportunity in future to explore more subtle ways to 
use visual material to represent a range of ideas or scale of interest, since 
any sequence of images might be combined to provide a progressive 
scale of options. 
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During the development of content and technical systems we needed ways 
to share concepts and explore their implications. Previously Blythe and 
Wright" had developed the use of literary pastiches in the style of 
well-known authors to create recognisable scenarios. T/;e Simpsons 
characters which were introduced explain some early technical scenarios, 
borrowed from a previous presentation which used them to anonymise 
participants in confidential industrial research."' 

We recognised that the Simpsons provided a rich cast of well-developed 
characters and experiences and one of Mark Blythe's roles in the project 
was to create short stories that showed how the different characters miglit 
react when confronted with aspects of the proposed exhibition. 



Voice: Meeting the Dauphin at Chinon castle, she conquered his 

skepticism as to her divine mission. In September 1429, Joan 
unsuccessfully besieged Paiis. The following spring she went to 
relieve Compiegne, but she was captured by the Burgundians and 
sold to the English, who were eager to destroy her influence by 
putting her to death. 

Lisa: Oh Mom! I'm learning! 

Marge comes over looking concerned and alarmed 

i\1arge: What about? 

Lisa: Joan of Arc 

Voice: She was condemned to life imprisonment. Shortly afterward, 
however, she was burned at the st - 

Marge drags Lisa away 

Lisa: Mom I was listening to that! 

iUlarge: What? It was the end. She was pardoned. They lived happily ever 
after. Come on Lisa, I forgot to open a window in the car; your 
Father will be starting to overheat. 



Figure 3, Example of Pastiche Scenario 

The resulting exhibition allowed us to explore how people deal with 
such interactions and reflect on our experience of designing for it. 
Having developed a working tool it was necessary to make it available for 
future research. Following the exhibition and some further work on how 
to incorporate 'invisible' LCDs, we redesigned the interactive exhibit as 
a portable system of 5 'black boxes' that allow a wide variety of digital 
content. We are now working with curators and designers to evaluate 
how they miglit use such a system and design for it, using an oral history 
archive from English Heritage as prospective content. 
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All this ambitious and complex development effort meant that the 
fieldwork had to be compressed into a relatively short period, more so 
because there were several changes in the Royal Armouries timetable 
completely outside our control. Real world events create very inflexible 
laboratories. 

New Knowledge and Understanding 

We have learned a lot about how to design for unencumbered, tacit 
interaction and about working with complex interdisciplinary problems. 
We have learned a little about how people respond to this kind of 
interaction. This was a very ambitious project and we have learned 
some practical lessons about conducting such research. Finally we have 
developed ideas about further work to clarify and develop these techniques 
and built tools to help with that further work. 

These new media require new kinds of designers. The design challenge 
integrated a variety of digital hardware and software with more 
conventional, audio, lighting and 3-dimensional thinking. When recruiting 
researchers it was evident that many design graduates have rather limited 
concepts of their roles, not recognising the challenges we faced, but some 
artists had developed relevant eclectic sets of practical knowledge and 
skills combined with an understanding of the experiential aspects of 
the work. Our more traditional software engineer provided a vital sense 
of structure to the project to balance the individualism of the artists. 
The core design/prototyping team of researchers, investigators and 
technicians finally included a software engineer; an artist/programmer 
with experience of installing electro-mechanical devices in artworks; 
another artist with extensive experience as a scene builder in the film 
industry; two furniture design graduates with experience of computer 
numerically controlled (CNC) production, vital for building custom 
electronic displays; film/video-makers and a human -computer interaction 
(HCI) researcher with an active interest in audio recording and editing. 

This interdisciplinary mix creates problems with both practice and 
expectations. We needed a balance between our various professional 
expectations, and what is realistic in practice, to carry out the research. 
Some of our audio content lacked the polish of the professionally 
produced video, but ensured a sufficient range of material for the research 
to investigate personalisation. The participants' coloured badges had a 
lashup quality as they were in development right up to the last minute. 
We had tension between some who saw the data from the project as its 
primary outcome, and others who felt it necessary to embody the work in 
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a lasting 'product' to support further research and demonstrations. 
On the other hand, our experience of past interdisciphnary debate 
indicates risk of mutual misunderstandings of terminology; however, 
personal relationships developed during the project and preceding 
network, which seemed to allow people to see intended meanings rather 
than misreading through a disciplinary filter. 

The blurring of roles and exchange of skills and ideas in the practical 
work seemed a very positive aspect of the project. We have observed 
similar effects in industry, where sharing new digital working media 
between different disciplines has fostered convergence of roles 
and understanding. 

The central concept of 'values' provided an essential focus for the 
development work, allowing an iterative process that moved between 
refining strategy and developing content. Users recognised the idea of 
'unspoken' values at work. Where the value concepts were more explicit, 
in the initial profiling exercise, it was clear that some ideas were easily 
recognised, but others were difficult for the users to interpret, indicating a 
need for more work on how to balance explicit and tacit judgements and 
evaluate the profiling tools in development. Since our concept of values 
is rather more open and fluid than previous research in cultural studies, 
this points to a promising area of 'experiential' knowledge which could be 
explored with our software and design approach. 

Finally, our fieldwork, although more limited than hoped due to technical 
challenges and external disruptions to the planned programme, showed 
that the audience found the personalisation approach and unencumbered 
interaction system very engaging. It promoted interaction between 
visitors; they found the interaction method easy to understand and adopt, 
and they enjoyed the way it worked and recognised its value. Those who 
had some historical expertise felt the approach was intrinsically more 
engaging and relevant than other approaches they had experienced. 
In some tests the technology failed to perform. When that happened 
participants were nevertheless intrigued by the system, understood it and 
engaged in thoughtful discussion about how it could work for them. 
One of the encouraging aspects of all this is that our approach appears to 
have achieved the aim of an ethical system, which operates subtly in the 
background but ensures that users are able to recognise its presence while 
interacting with it intuitively. 



